#1611 - Refactor SQL module to use PreparedStatement in SQLSpout and IndexerBolt #1766

rajucomp · 2025-12-27T15:29:18Z

This PR refactors parts of the SQL module to use PreparedStatement instead of SQL string concatenation, in accordance with issue #1611.

Thank you for contributing to Apache StormCrawler.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

For all changes

Is there a issue associated with this PR? Is it referenced in the commit message?
Does your PR title start with #XXXX where XXXX is the issue number you are trying to resolve?
Has your PR been rebased against the latest commit within the target branch (typically main)?
Is your initial contribution a single, squashed commit?
Is the code properly formatted with mvn git-code-format:format-code -Dgcf.globPattern="**/*" -Dskip.format.code=false?

For code changes

Have you ensured that the full suite of tests is executed via mvn clean verify?
Have you written or updated unit tests to verify your changes?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE file, including the main LICENSE file?
If applicable, have you updated the NOTICE file, including the main NOTICE file?

Note

Please ensure that once the PR is submitted, you check GitHub Actions for build issues and submit an update to your PR as soon as possible.

…olt for improved readability and performance

rajucomp · 2025-12-27T15:31:50Z

@jnioche Could you approve the CI please ? And Can i get review for this PR ? Curious to know your thoughts. Thanks!

…olt for improved readability and performance

rzo1 · 2025-12-27T19:59:30Z

Thanks for the PR. I have triggered the CI which results in

[INFO] Scanning classes for violations...
Error:  Forbidden method invocation: java.lang.String#format(java.lang.String,java.lang.Object[]) [Uses default locale]
Error:    in org.apache.stormcrawler.sql.StatusUpdaterBolt (StatusUpdaterBolt.java:107)
Error:  Forbidden method invocation: java.lang.String#format(java.lang.String,java.lang.Object[]) [Uses default locale]
Error:    in org.apache.stormcrawler.sql.StatusUpdaterBolt (StatusUpdaterBolt.java:114)
Error:  Forbidden method invocation: java.lang.String#format(java.lang.String,java.lang.Object[]) [Uses default locale]
Error:    in org.apache.stormcrawler.sql.SQLSpout (SQLSpout.java:125)
Error:  Forbidden method invocation: java.lang.String#format(java.lang.String,java.lang.Object[]) [Uses default locale]
Error:    in org.apache.stormcrawler.sql.SQLSpout (SQLSpout.java:234)
Error:  Forbidden method invocation: java.lang.String#format(java.lang.String,java.lang.Object[]) [Uses default locale]
Error:    in org.apache.stormcrawler.sql.IndexerBolt (IndexerBolt.java:175)
Error:  Forbidden method invocation: java.lang.String#format(java.lang.String,java.lang.Object[]) [Uses default locale]
Error:    in org.apache.stormcrawler.sql.IndexerBolt (IndexerBolt.java:178)
Error:  Scanned 6 class file(s) for forbidden API invocations (in 0.04s), 6 error(s).

Didn't look into the code diff yet.

rajucomp · 2025-12-27T20:44:20Z

@rzo1 Request to trigger the CI again.

rzo1

Thanks for the PR! I have some questions / suggestions and didn't looked deep into the diff yet:

Is it possible to move the prepared statements to the class level? For the StatusUpdaterBolt and the spout, the statements shouldn’t change per tuple, and since Storm bolts are single-threaded per instance, this should help reduce overhead.
I think this module is currently untested. It would be great to add some tests (maybe using test containers). I know the old implementation didn’t have tests either, but having them would help ensure everything works as expected.

rajucomp · 2026-01-01T18:37:34Z

@rzo1 Request to trigger the CI.

… up.

…ndexing functionality

rajucomp · 2026-01-02T12:01:27Z

@rzo1 I think the PR is in a good shape for review now. Tests have been added and prepared statements have been moved back to class level. Let me know your thoughts.

rzo1

Overall, lgtm.

external/sql/pom.xml

external/sql/src/main/java/org/apache/stormcrawler/sql/IndexerBolt.java

rzo1 · 2026-01-06T14:57:31Z

external/sql/src/main/java/org/apache/stormcrawler/sql/StatusUpdaterBolt.java

    }

    @Override
    public void cleanup() {


There is a slight risk that this method isn't called in a cluster setup:

The cleanup method is called when a Bolt is being shutdown and should cleanup any resources that were opened. There's no guarantee that this method will be called on the cluster: for example, if the machine the task is running on blows up, there's no way to invoke the method. The cleanup method is intended for when you run topologies in local mode (where a Storm cluster is simulated in a process), and you want to be able to run and kill many topologies without suffering any resource leaks.

Don't think it would be a huge issue for now.

I was thinking about this. We should look into making these interfaces extend AutoCloseable for better resources cleanup. This would also help to use these components with try-with-resources block. https://docs.oracle.com/javase/8/docs/api/java/lang/AutoCloseable.html.
Would like to hear your thoughts on this.

I see your point about making these interfaces extend AutoCloseable for better resource cleanup and enabling usage with try-with-resources. That approach makes sense in general for components that are manually managed and need explicit cleanup.

However, in the case of a Storm bolt, it doesn’t quite fit. Bolts are executed by the Storm runtime, and users typically don’t instantiate or manage them directly in a try-with-resources block. Implementing AutoCloseable here wouldn’t provide any practical benefit, since resource management is handled by Storm itself rather than the user.

pom.xml

rajucomp · 2026-01-07T18:55:41Z

@rzo1 The comments have been addressed. Let me know your thoughts.

rajucomp · 2026-01-09T19:40:55Z

@rzo1 Sorry for asking this stupid question but are we waiting for more approvals to merge the PR ?

rzo1 · 2026-01-09T19:52:37Z

It’s not a stupid question.

Typically, we wait at least 72 hours so that people in different time zones have a chance to review it (the same applies to release votes sent to the dev@ mailing list - if you’re interested in SC, feel free to subscribe).

This period can be shorter if necessary. Also, keep in mind that most committers contribute in their spare time and are not paid to work on SC full-time, so reviews and merges Usually may take some additional time.

rzo1 · 2026-01-10T16:51:24Z

@rajucomp Wondering if you did run a test crawl with this code? Otherwise, we should do a runtime validation before integrating it.
(If not, you can use https://github.com/rzo1/sc-warc-demo as a starter and replace the warc part with the sql module)

rajucomp · 2026-01-11T14:39:29Z

Hi @rzo1. Followed the steps. Here is the output

1. Infrastructure Status

All Docker containers are running and healthy:

NAMES         STATUS                    PORTS  
supervisor    Up 20 minutes  
ui            Up 20 minutes             127.0.0.1:8080->8080/tcp  
nimbus        Up 20 minutes  
zookeeper     Up 20 minutes             2181/tcp, 2888/tcp, 3888/tcp, 8080/tcp  
mysql         Up 20 minutes (healthy)   127.0.0.1:3306->3306/tcp  
urlfrontier   Up 20 minutes             127.0.0.1:7071->7071/tcp

Key Points:

MySQL container shows (healthy) status indicating successful health checks
All Storm components (nimbus, supervisor, ui) are operational
URLFrontier service is accessible on port 7071

2. Storm Worker Logs - URL Fetching & SQL Persistence

2.1 URLFrontier Spout Initialization

2026-01-11 14:11:37.397 o.a.s.e.s.SpoutExecutor Thread-23-spout-executor[9, 9] [INFO] Opening spout spout:[9]  
2026-01-11 14:11:37.425 o.a.s.u.ManagedChannelUtil Thread-23-spout-executor[9, 9] [INFO] Initialisation of connection to URLFrontier service on urlfrontier:7071  
2026-01-11 14:11:37.540 o.a.s.u.Spout Thread-23-spout-executor[9, 9] [INFO] Initialized URLFrontier Spout without crawlId  
2026-01-11 14:11:37.541 o.a.s.e.s.SpoutExecutor Thread-23-spout-executor[9, 9] [INFO] Opened spout spout:[9]  
2026-01-11 14:11:37.541 o.a.s.e.s.SpoutExecutor Thread-23-spout-executor[9, 9] [INFO] Activating spout spout:[9]

Evidence: URLFrontier Spout successfully connected to the URLFrontier service.

2.2 URL Fetching (FetcherBolt)

# Initial seed URL fetch - HTTP 308 redirect  
2026-01-11 14:11:38.736 o.a.s.b.FetcherBolt FetcherThread #31 [INFO] [Fetcher #4] Fetched https://kodis.iao.fraunhofer.de with status 308 in msec 69  
  
# Successful page fetch - HTTP 200  
2026-01-11 14:17:07.143 o.a.s.b.FetcherBolt FetcherThread #44 [INFO] [Fetcher #4] Fetched https://www.kodis.iao.fraunhofer.de/ with status 200 in msec 315  
  
# Subsequent fetches showing continuous crawling  
2026-01-11 14:22:08.262 o.a.s.b.FetcherBolt FetcherThread #0 [INFO] [Fetcher #4] Fetched https://www.kodis.iao.fraunhofer.de/ with status 200 in msec 282  
2026-01-11 14:27:09.316 o.a.s.b.FetcherBolt FetcherThread #19 [INFO] [Fetcher #4] Fetched https://www.kodis.iao.fraunhofer.de/ with status 200 in msec 290

Evidence: FetcherBolt successfully fetching URLs with proper HTTP status codes.

2.3 HTML Parsing (JSoupParserBolt)

2026-01-11 14:17:07.148 o.a.s.b.JSoupParserBolt Thread-15-parse-executor[6, 6] [INFO] Parsing : starting https://www.kodis.iao.fraunhofer.de/  
2026-01-11 14:17:07.244 o.a.s.b.JSoupParserBolt Thread-15-parse-executor[6, 6] [INFO] Parsed https://www.kodis.iao.fraunhofer.de/ in 50 msec  
2026-01-11 14:17:07.252 o.a.s.b.JSoupParserBolt Thread-15-parse-executor[6, 6] [INFO] Total for https://www.kodis.iao.fraunhofer.de/ - 58 msec

Evidence: JSoupParserBolt successfully parsing HTML content.

2.4 Content Indexing (StdOutIndexer)

2026-01-11 14:17:07.253 o.a.s.i.StdOutIndexer Thread-20-index-executor[5, 5] [INFO] content     561 chars  
2026-01-11 14:17:07.253 o.a.s.i.StdOutIndexer Thread-20-index-executor[5, 5] [INFO] url         https://www.kodis.iao.fraunhofer.de/  
2026-01-11 14:17:07.253 o.a.s.i.StdOutIndexer Thread-20-index-executor[5, 5] [INFO] keywords    IAO Kodis  
2026-01-11 14:17:07.253 o.a.s.i.StdOutIndexer Thread-20-index-executor[5, 5] [INFO] domain      fraunhofer.de  
2026-01-11 14:17:07.253 o.a.s.i.StdOutIndexer Thread-20-index-executor[5, 5] [INFO] format      html  
2026-01-11 14:17:07.253 o.a.s.i.StdOutIndexer Thread-20-index-executor[5, 5] [INFO] description 137 chars  
2026-01-11 14:17:07.253 o.a.s.i.StdOutIndexer Thread-20-index-executor[5, 5] [INFO] title       Fraunhofer IAO - KODIS: Forschungs- und Innovationszentrum Kognitive Dienstleistungssysteme

Evidence: Content successfully extracted and indexed with metadata.

2.5 SQL StatusUpdaterBolt - Batch Persistence

# Single URL insert (redirect status)  
2026-01-11 14:11:41.498 o.a.s.s.StatusUpdaterBolt pool-9-thread-1 [INFO] About to execute batch - triggered by time. Due 1768140700807, now 1768140701498  
2026-01-11 14:11:41.504 o.a.s.s.StatusUpdaterBolt pool-9-thread-1 [INFO] Batched 1 inserts executed in 5 msec  
  
# Batch insert of discovered URLs (27 URLs from parsed page)  
2026-01-11 14:17:09.501 o.a.s.s.StatusUpdaterBolt pool-9-thread-1 [INFO] About to execute batch - triggered by time. Due 1768141029256, now 1768141029501  
2026-01-11 14:17:09.520 o.a.s.s.StatusUpdaterBolt pool-9-thread-1 [INFO] Batched 27 inserts executed in 18 msec

Evidence: SQL StatusUpdaterBolt successfully persisting URL statuses to MySQL in batches.

3. MySQL Database - Direct Verification

3.1 Database Connection & Schema Verification

sh-5.1# mysql -u root -prootpassword  
mysql: [Warning] Using a password on the command line interface can be insecure.  
Welcome to the MySQL monitor.  Commands end with ; or \g.  
Your MySQL connection id is 134  
Server version: 8.0.44 MySQL Community Server - GPL  
  
mysql> SHOW DATABASES;  
+--------------------+  
| Database           |  
+--------------------+  
| crawl              |  
| information_schema |  
| mysql              |  
| performance_schema |  
| sys                |  
+--------------------+  
5 rows in set (0.00 sec)  
  
mysql> USE crawl;  
Database changed  
  
mysql> SHOW TABLES;  
+-----------------+  
| Tables_in_crawl |  
+-----------------+  
| content         |  
| metrics         |  
| urls            |  
+-----------------+  
3 rows in set (0.01 sec)

Evidence: Database crawl exists with all required tables (urls, metrics, content).

3.2 URL Status Distribution

mysql> SELECT status, COUNT(*) as count FROM urls GROUP BY status;  
+-------------+-------+  
| status      | count |  
+-------------+-------+  
| DISCOVERED  |    29 |  
| REDIRECTION |     1 |  
+-------------+-------+  
2 rows in set (0.00 sec)  
  
mysql> SELECT COUNT(*) as total_urls FROM urls;  
+------------+  
| total_urls |  
+------------+  
|         30 |  
+------------+  
1 row in set (0.00 sec)

Evidence: 30 URLs tracked in database - 29 discovered, 1 redirection.

3.3 Complete URLs Table Data

mysql> SELECT * FROM urls;  
+----------------------------------------------------------------------------------------------------------------------------------+-------------+---------------------+--------------------------------------------------------+--------+-----------------------------+  
| url                                                                                                                              | status      | nextfetchdate       | metadata                                               | bucket | host                        |  
+----------------------------------------------------------------------------------------------------------------------------------+-------------+---------------------+--------------------------------------------------------+--------+-----------------------------+  
| https://kodis.iao.fraunhofer.de                                                                                                  | REDIRECTION | 2026-01-12 14:26:41 | _redirTo=https://www.kodis.iao.fraunhofer.de/          |      0 | kodis.iao.fraunhofer.de     |  
| https://publica.fraunhofer.de/entities/publication/253205a3-ed71-4926-b26d-b07bca516800/details                                  | DISCOVERED  | 2026-01-11 14:17:07 | url.path=https://www.kodis.iao.fraunhofer.de/  depth=1 |      0 | publica.fraunhofer.de       |  
| https://www.iao.fraunhofer.de/                                                                                                   | DISCOVERED  | 2026-01-11 14:17:07 | url.path=https://www.kodis.iao.fraunhofer.de/  depth=1 |      0 | www.iao.fraunhofer.de       |  
| https://www.kodis.iao.fraunhofer.de/                                                                                             | DISCOVERED  | 2026-01-11 14:11:39 | url.path=https://kodis.iao.fraunhofer.de       depth=1 |      0 | www.kodis.iao.fraunhofer.de |  
| https://www.kodis.iao.fraunhofer.de/de/aktuelles.html                                                                            | DISCOVERED  | 2026-01-11 14:17:07 | url.path=https://www.kodis.iao.fraunhofer.de/  depth=1 |      0 | www.kodis.iao.fraunhofer.de |  
| https://www.kodis.iao.fraunhofer.de/de/leistungen.html                                                                           | DISCOVERED  | 2026-01-11 14:17:07 | url.path=https://www.kodis.iao.fraunhofer.de/  depth=1 |      0 | www.kodis.iao.fraunhofer.de |  
| https://www.kodis.iao.fraunhofer.de/de/projekte.html                                                                             | DISCOVERED  | 2026-01-11 14:17:07 | url.path=https://www.kodis.iao.fraunhofer.de/  depth=1 |      0 | www.kodis.iao.fraunhofer.de |  
| https://www.kodis.iao.fraunhofer.de/de/publikationen.html                                                                        | DISCOVERED  | 2026-01-11 14:17:07 | url.path=https://www.kodis.iao.fraunhofer.de/  depth=1 |      0 | www.kodis.iao.fraunhofer.de |  
| https://www.kodis.iao.fraunhofer.de/de/ueber-uns.html                                                                            | DISCOVERED  | 2026-01-11 14:17:07 | url.path=https://www.kodis.iao.fraunhofer.de/  depth=1 |      0 | www.kodis.iao.fraunhofer.de |  
| https://www.kodis.iao.fraunhofer.de/de/veranstaltungen.html                                                                      | DISCOVERED  | 2026-01-11 14:17:07 | url.path=https://www.kodis.iao.fraunhofer.de/  depth=1 |      0 | www.kodis.iao.fraunhofer.de |  
| https://www.kodis.iao.fraunhofer.de/en.html                                                                                      | DISCOVERED  | 2026-01-11 14:17:07 | url.path=https://www.kodis.iao.fraunhofer.de/  depth=1 |      0 | www.kodis.iao.fraunhofer.de |  
+----------------------------------------------------------------------------------------------------------------------------------+-------------+---------------------+--------------------------------------------------------+--------+-----------------------------+  
(30 rows total - truncated for brevity)

Key Observations:

status column correctly stores URL states (DISCOVERED, REDIRECTION)
nextfetchdate shows scheduled refetch times
metadata preserves crawl path and depth information
host column enables per-host politeness scheduling

4. SQL MetricsConsumer - Metrics Table

mysql> SELECT * FROM metrics ORDER BY timestamp DESC LIMIT 15;  
+------+----------------+-----------+--------------+----------------+-----------------------------------------------+-------+---------------------+  
| id   | srcComponentId | srcTaskId | srcWorkerHost| srcWorkerPort  | name                                          | value | timestamp           |  
+------+----------------+-----------+--------------+----------------+-----------------------------------------------+-------+---------------------+  
| 2189 | fetcher        |         4 | cbead6cecad0 |           6700 | in_queues                                     |     0 | 2026-01-11 14:27:38 |  
| 2188 | fetcher        |         4 | cbead6cecad0 |           6700 | fetcher_average_persec.fetched_perSec         |     0 | 2026-01-11 14:27:38 |  
| 2187 | fetcher        |         4 | cbead6cecad0 |           6700 | fetcher_average_persec.bytes_fetched_perSec   |     0 | 2026-01-11 14:27:38 |  
| 2186 | fetcher        |         4 | cbead6cecad0 |           6700 | activethreads                                 |     0 | 2026-01-11 14:27:38 |  
| 2185 | fetcher        |         4 | cbead6cecad0 |           6700 | fetcher_counter.fetched                       |     0 | 2026-01-11 14:27:38 |  
| 2184 | fetcher        |         4 | cbead6cecad0 |           6700 | fetcher_counter.robots.fromCache              |     0 | 2026-01-11 14:27:38 |  
| 2183 | fetcher        |         4 | cbead6cecad0 |           6700 | fetcher_counter.bytes_fetched                 |     0 | 2026-01-11 14:27:38 |  
| 2182 | fetcher        |         4 | cbead6cecad0 |           6700 | fetcher_counter.robots.fetched                |     0 | 2026-01-11 14:27:38 |  
| 2181 | fetcher        |         4 | cbead6cecad0 |           6700 | fetcher_counter.status_200                    |     0 | 2026-01-11 14:27:38 |  
| 2180 | fetcher        |         4 | cbead6cecad0 |           6700 | fetcher_counter.status_308                    |     0 | 2026-01-11 14:27:38 |  
| 2179 | fetcher        |         4 | cbead6cecad0 |           6700 | num_queues                                    |     0 | 2026-01-11 14:27:38 |  
| 2178 | spout          |         9 | cbead6cecad0 |           6700 | inPurgatory                                   |     2 | 2026-01-11 14:27:38 |  
| 2177 | spout          |         9 | cbead6cecad0 |           6700 | numQueues                                     |     0 | 2026-01-11 14:27:38 |  
| 2176 | spout          |         9 | cbead6cecad0 |           6700 | beingProcessed                                |     0 | 2026-01-11 14:27:38 |  
| 2175 | spout          |         9 | cbead6cecad0 |           6700 | buffer_size                                   |     0 | 2026-01-11 14:27:38 |  
+------+----------------+-----------+--------------+----------------+-----------------------------------------------+-------+---------------------+

Evidence: SQL MetricsConsumer successfully writing crawler metrics to MySQL including:

Fetcher statistics (fetched count, bytes, HTTP status codes)
Spout metrics (queue sizes, purgatory count)
Worker identification (host, port, component)

5. URLFrontier Statistics

$ java -cp target/stormcrawler-sql-1.0-SNAPSHOT.jar \  
 crawlercommons.urlfrontier.client.Client -t localhost -p 7071 GetStats  
Number of queues: 2  
Active URLs: 2  
In process: 2  
active_queues = 2  
completed = 0  
  
$ java -cp target/stormcrawler-sql-1.0-SNAPSHOT.jar \  
 crawlercommons.urlfrontier.client.Client -t localhost -p 7071 ListQueues  
kodis.iao.fraunhofer.de  
www.kodis.iao.fraunhofer.de

Evidence: URLFrontier managing 2 queues (one per host) with active URL processing.

6. Summary

Component	Status	Evidence
Storm Topology	✅ Running	`sql-crawler-1-1768140688` deployed and active
URLFrontier Spout	✅ Working	Connected to `urlfrontier:7071`, 2 queues active
FetcherBolt	✅ Working	Fetched URLs with HTTP 200/308 responses
JSoupParserBolt	✅ Working	Parsed pages, extracted content in ~50ms
SQL StatusUpdaterBolt	✅ Working	Batch inserts (1-27 URLs) in 2-18ms
SQL MetricsConsumer	✅ Working	2189+ metrics records in `metrics` table
MySQL Persistence	✅ Working	30 URLs tracked with status in `urls` table
Docker Infrastructure	✅ Healthy	All 6 containers running, MySQL health check passing

7. Configuration Reference

SQL Connection Settings (from crawler-conf.yaml)

sql.connection:  
 url: "jdbc:mysql://mysql:3306/crawl?useSSL=false&allowPublicKeyRetrieval=true&serverTimezone=UTC" user: "crawler" password: "crawler" rewriteBatchedStatements: true useBatchMultiSend: true  
sql.status.table: "urls"  
sql.metrics.table: "metrics"

Key Topology Components (from crawler.flux)

Spout: crawlercommons.urlfrontier.stormcrawler.Spout (URLFrontier integration)
Status Bolt: org.apache.stormcrawler.sql.StatusUpdaterBolt (SQL persistence)
Metrics Consumer: org.apache.stormcrawler.sql.metrics.MetricsConsumer

Conclusion: The StormCrawler SQL module integration is fully functional. URLs are being fetched, parsed, and their statuses are correctly persisted to MySQL. The MetricsConsumer is recording crawler performance data, and the URLFrontier is managing URL scheduling across multiple host queues.

There were some changes required in the config file. So, I create a fork of your repo and update the configs at https://github.com/rajucomp/sc-sql--warc-demo. This will help in future testing efforts.

Also attached a screenshot of the storm UI running. Not sure if further verifications are needed.

Let me know your thoughts. Thanks!

rzo1

Thanks for the PR and your contribution.

Refactor SQL module to use PreparedStatement in SQLSpout and IndexerB…

dbe1be8

…olt for improved readability and performance

Refactor SQL module to use PreparedStatement in SQLSpout and IndexerB…

5ff026e

…olt for improved readability and performance

rzo1 requested review from jnioche and sigee December 27, 2025 19:55

Fix forbidden violations

03916ef

rzo1 reviewed Dec 29, 2025

View reviewed changes

raju.gupta added 5 commits January 1, 2026 16:45

changes following PR review. Added tests for SQLSpout

4e37c55

changes following PR review. Added tests for SQLSpout

42dd9f3

changes following PR review. Added tests for SQLSpout

37d595f

changes following PR review. Added tests for StatusUpdaterBolt

747aa9b

Add test dependencies and update MySQL container version in tests

a69212d

raju.gupta added 15 commits January 1, 2026 20:45

Fix falkiness in testStoreDiscoveredURL

3e3d6bd

ScheduledExecutorService should be shit down when the bolt is cleaned…

9245499

… up.

Refactor IndexerBolt to improve code clarity and add unit tests for i…

fb7febe

…ndexing functionality

Introduce AbstractSQLTest for shared MySQL container setup in SQL tests

c14419b

Introduce AbstractSQLTest for shared MySQL container setup in SQL tests

051f0d9

Introduce AbstractSQLTest for shared MySQL container setup in SQL tests

28a8596

Introduce AbstractSQLTest for shared MySQL container setup in SQL tests

c43e08f

Introduce AbstractSQLTest for shared MySQL container setup in SQL tests

bf44537

Introduce AbstractSQLTest for shared MySQL container setup in SQL tests

5e47e81

Introduce AbstractSQLTest for shared MySQL container setup in SQL tests

a1f3691

Introduce AbstractSQLTest for shared MySQL container setup in SQL tests

0a6ec5c

Refactor StatusUpdaterBoltTest to improve setup and cleanup methods

43658c2

Undo changes in MetricsConsumer

86e9f3a

minor changes

f7859ee

minor changes

190efb2

rajucomp requested a review from rzo1 January 3, 2026 22:21

rzo1 reviewed Jan 6, 2026

View reviewed changes

rajucomp and others added 3 commits January 7, 2026 18:43

Merge branch 'apache:main' into main

9e37c1c

Changes following PR review.

b8f1757

Changes following PR review.

5ef9728

rajucomp requested a review from rzo1 January 7, 2026 18:55

rzo1 approved these changes Jan 7, 2026

View reviewed changes

rzo1 added this to the 3.5.1 milestone Jan 10, 2026

rzo1 added the SQL label Jan 10, 2026

rzo1 changed the title ~~Refactor SQL module to use PreparedStatement in SQLSpout and IndexerBolt #1611~~ #1611 - Refactor SQL module to use PreparedStatement in SQLSpout and IndexerBolt Jan 10, 2026

rzo1 approved these changes Jan 11, 2026

View reviewed changes

rzo1 merged commit e222c7d into apache:main Jan 11, 2026
2 checks passed

#1611 - Refactor SQL module to use PreparedStatement in SQLSpout and IndexerBolt #1766

#1611 - Refactor SQL module to use PreparedStatement in SQLSpout and IndexerBolt #1766

Uh oh!

Conversation

rajucomp commented Dec 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

For all changes

For code changes

Note

Uh oh!

rajucomp commented Dec 27, 2025

Uh oh!

rzo1 commented Dec 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rajucomp commented Dec 27, 2025

Uh oh!

rzo1 left a comment

Choose a reason for hiding this comment

Uh oh!

rajucomp commented Jan 1, 2026

Uh oh!

rajucomp commented Jan 2, 2026

Uh oh!

rzo1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

rzo1 Jan 6, 2026

Choose a reason for hiding this comment

Uh oh!

rajucomp Jan 7, 2026

Choose a reason for hiding this comment

Uh oh!

rzo1 Jan 7, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rajucomp commented Jan 7, 2026

Uh oh!

rajucomp commented Jan 9, 2026

Uh oh!

rzo1 commented Jan 9, 2026

Uh oh!

rzo1 commented Jan 10, 2026

Uh oh!

rajucomp commented Jan 11, 2026

1. Infrastructure Status

2. Storm Worker Logs - URL Fetching & SQL Persistence

2.1 URLFrontier Spout Initialization

2.2 URL Fetching (FetcherBolt)

2.3 HTML Parsing (JSoupParserBolt)

2.4 Content Indexing (StdOutIndexer)

2.5 SQL StatusUpdaterBolt - Batch Persistence

3. MySQL Database - Direct Verification

3.1 Database Connection & Schema Verification

3.2 URL Status Distribution

3.3 Complete URLs Table Data

4. SQL MetricsConsumer - Metrics Table

5. URLFrontier Statistics

6. Summary

7. Configuration Reference

SQL Connection Settings (from crawler-conf.yaml)

Key Topology Components (from crawler.flux)

Uh oh!

rzo1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

rajucomp commented Dec 27, 2025 •

edited

Loading

rzo1 commented Dec 27, 2025 •

edited

Loading